Colorless green recurrent networks dream hierarchically
نویسندگان
چکیده
Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-trivial properties of language. We investigate here to what extent RNNs learn to track abstract hierarchical syntactic structure. We test whether RNNs trained with a generic language modeling objective in four languages (Italian, English, Hebrew, Russian) can predict long-distance number agreement in various constructions. We include in our evaluation nonsensical sentences where RNNs cannot rely on semantic or lexical cues (“The colorless green ideas i s ea i I ate with the chair sleep l s ee l furiously”), and, for Italian, we compare model performance to human intuitions. Our language-model-trained RNNs make reliable predictions about long-distance agreement, and do not lag much behind human performance. We thus bring support to the hypothesis that RNNs are not just shallowpattern extractors, but they also acquire deeper grammatical competence.
منابع مشابه
Recurrent Autoassociative Networks: Developing Distributed Representations of Hierarchically Structured Sequences by Autoassociation
متن کامل
Colorless polydopamine coatings for creating functional interfaces
Since the emergence of surface modification techniques, including polydopamine (PDA) layers using a single deposition process that is based on the oxidative self-polymerization of dopamine (DA), the number of studies concerning the functionalization of material surfaces via PDA coating has increased. While PDA is simple to apply to cover onto various kinds of substrates, most of conventional PD...
متن کاملRobust stability of stochastic fuzzy impulsive recurrent neural networks with\ time-varying delays
In this paper, global robust stability of stochastic impulsive recurrent neural networks with time-varyingdelays which are represented by the Takagi-Sugeno (T-S) fuzzy models is considered. A novel Linear Matrix Inequality (LMI)-based stability criterion is obtained by using Lyapunov functional theory to guarantee the asymptotic stability of uncertain fuzzy stochastic impulsive recurrent neural...
متن کاملColorless Green Ideas Sleep Furiously Revisited: A Statistical Perspective
In the present study we provide empirical evidence that human learners succeed in an artificial-grammar learning task that involves recognizing grammatical sequences whose bigram frequencies from the training corpus are zero. This result begs explanation: Whatever strategy is being used to perform the task, it cannot rely on the simple cooccurrence of elements in the training corpus. While rule...
متن کاملGenome Dreaming
We investigate and build the framework required to generate or “dream” of genomes that encode novel organisms. Importantly, we find that developing methods to validate the generated sequences is as important as building a good model to realize this task due to the lack of methods capable of interpreting and evaluating a generated sequence. We report on our results in building and optimizing a g...
متن کامل